Fine-grained OS Behavior Characterization

Monolithic operating systems (OSes) are complex pieces of software that usually offer very little reliability and security guarantees. Faulty user-space applications can generally be restarted without affecting the existing concurrent communications but those involving the faulty processes. On the other hand, in a monolithic OS design, the kernel and all its components share a common address space and any component can potentially invoke any kernel function. In this scenario, it becomes extremely complicated—if not impossible—to isolate and restart faulty kernel components as it is generally hard to define their boundaries and interactions (e.g., what kernel control paths are executed and how information is shared). Unfortunately, run-time bugs are not the only security threats an OS must deal with. For instance, malicious components may undermine the security of the whole system from its root. Kernel rootkits can be installed on the system to replace or modify the legitimate behavior of arbitrary subsystems of the OS. It is under this perspective that it becomes clear we need techniques that enable us to characterize and describe the behavior of the whole OS, i.e., the kernel and its components. The research community has so far proposed a number of approaches aimed at characterizing the behaviors of user-space processes. Although each one different from the other, such techniques generally rely on the same underlying intuition, i.e., the behavior of a process can be expressed by the sequence of system calls the process invokes [1]. Unfortunately, such an intuition easily drops when monolithic OSes are considered, because well-defined and easy-to-enforce communication interfaces among kernel subsystems are generally missing. While promising, recent attempts to characterize the behavior of kernel OS can be indeed quite complicated and currently confined to malware analysts rather than on-line behavior-based policy enforcement for end-user systems. For instance, the completeness and scalability of the analysis proposed in [3] depend on the particular selection of kernel execution paths and well-defined knowledge on what can be considered sensitive. It is our belief that a microkernel and multiserver OS design offers better opportunities to characterize and describe the behavior of the whole operating system. Recent studies have shown that program-centric analyses approaches may not be well-suited to describe the behavior of generic userspace processes, as they may not generalize well, especially when the considered applications are exposed to a plethora of previously-unseen and realistic input [2]. In such scenar-